Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1190 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.3262. Min Act: -0.0000

Data Index: 755549 (C4 (Web Text))

Max Activating Token Index: 924

Text #1

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.3026. Min Act: -0.0000

Data Index: 75505 (C4 (Web Text))

Max Activating Token Index: 429

Text #2

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2889. Min Act: -0.0000

Data Index: 569853 (C4 (Web Text))

Max Activating Token Index: 485

Text #3

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2846. Min Act: -0.0000

Data Index: 236217 (C4 (Web Text))

Max Activating Token Index: 449

Text #4

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2499. Min Act: -0.0000

Data Index: 1350320 (C4 (Web Text))

Max Activating Token Index: 828

Text #5

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2843. Min Act: -0.0000

Data Index: 136986 (C4 (Web Text))

Max Activating Token Index: 363

Text #6

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2732. Min Act: -0.0000

Data Index: 569658 (C4 (Web Text))

Max Activating Token Index: 363

Text #7

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2925. Min Act: -0.0000

Data Index: 18247 (C4 (Web Text))

Max Activating Token Index: 644

Text #8

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2681. Min Act: -0.0000

Data Index: 205637 (C4 (Web Text))

Max Activating Token Index: 321

Text #9

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2757. Min Act: -0.0000

Data Index: 40860 (C4 (Web Text))

Max Activating Token Index: 872

Text #10

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2539. Min Act: -0.0000

Data Index: 1314403 (C4 (Web Text))

Max Activating Token Index: 139

Text #11

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2641. Min Act: -0.0000

Data Index: 1152859 (C4 (Web Text))

Max Activating Token Index: 34

Text #12

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2460. Min Act: -0.0000

Data Index: 371291 (C4 (Web Text))

Max Activating Token Index: 926

Text #13

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2495. Min Act: -0.0000

Data Index: 417227 (C4 (Web Text))

Max Activating Token Index: 764

Text #14

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2585. Min Act: -0.0000

Data Index: 1002080 (C4 (Web Text))

Max Activating Token Index: 135

Text #15

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2443. Min Act: -0.0000

Data Index: 1151611 (C4 (Web Text))

Max Activating Token Index: 325

Text #16

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2438. Min Act: -0.0000

Data Index: 644439 (C4 (Web Text))

Max Activating Token Index: 517

Text #17

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2418. Min Act: -0.0000

Data Index: 851053 (C4 (Web Text))

Max Activating Token Index: 203

Text #18

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2343. Min Act: -0.0000

Data Index: 1080154 (C4 (Web Text))

Max Activating Token Index: 136

Text #19

Max Range: 0.3262. Min Range: -0.3262

Max Act: 0.2383. Min Act: -0.0000

Data Index: 485302 (C4 (Web Text))

Max Activating Token Index: 358