Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 429 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.6230. Min Act: -0.0000

Data Index: 190519 (C4 (Web Text))

Max Activating Token Index: 916

Text #1

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1697. Min Act: -0.0000

Data Index: 225943 (C4 (Web Text))

Max Activating Token Index: 513

Text #2

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.2344. Min Act: -0.0000

Data Index: 610577 (C4 (Web Text))

Max Activating Token Index: 705

Text #3

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.1313. Min Act: -0.0000

Data Index: 825996 (C4 (Web Text))

Max Activating Token Index: 129

Text #4

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0223. Min Act: -0.0000

Data Index: 485090 (C4 (Web Text))

Max Activating Token Index: 63

Text #5

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0555. Min Act: -0.0000

Data Index: 1078563 (C4 (Web Text))

Max Activating Token Index: 708

Text #6

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0270. Min Act: -0.0000

Data Index: 828288 (C4 (Web Text))

Max Activating Token Index: 161

Text #7

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9868. Min Act: -0.0000

Data Index: 1049825 (C4 (Web Text))

Max Activating Token Index: 840

Text #8

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9423. Min Act: -0.0000

Data Index: 879829 (C4 (Web Text))

Max Activating Token Index: 907

Text #9

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0252. Min Act: -0.0000

Data Index: 1427513 (Python Code)

Max Activating Token Index: 577

Text #10

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9599. Min Act: -0.0000

Data Index: 760027 (C4 (Web Text))

Max Activating Token Index: 31

Text #11

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.8706. Min Act: -0.0000

Data Index: 773235 (C4 (Web Text))

Max Activating Token Index: 157

Text #12

Max Range: 1.6230. Min Range: -1.6230

Max Act: 1.0001. Min Act: -0.0000

Data Index: 107907 (C4 (Web Text))

Max Activating Token Index: 212

Text #13

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.8662. Min Act: -0.0000

Data Index: 995732 (C4 (Web Text))

Max Activating Token Index: 119

Text #14

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9131. Min Act: -0.0000

Data Index: 351093 (C4 (Web Text))

Max Activating Token Index: 257

Text #15

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9759. Min Act: -0.0000

Data Index: 807834 (C4 (Web Text))

Max Activating Token Index: 814

Text #16

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9623. Min Act: -0.0000

Data Index: 224629 (C4 (Web Text))

Max Activating Token Index: 363

Text #17

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9135. Min Act: -0.0000

Data Index: 611619 (C4 (Web Text))

Max Activating Token Index: 312

Text #18

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9048. Min Act: -0.0000

Data Index: 943541 (C4 (Web Text))

Max Activating Token Index: 682

Text #19

Max Range: 1.6230. Min Range: -1.6230

Max Act: 0.9395. Min Act: -0.0000

Data Index: 765978 (C4 (Web Text))

Max Activating Token Index: 158