Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2625 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0424. Min Range: -1.0424

Max Act: 1.0424. Min Act: -0.0000

Data Index: 848667 (C4 (Web Text))

Max Activating Token Index: 884

Text #1

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.8435. Min Act: -0.0000

Data Index: 1327853 (C4 (Web Text))

Max Activating Token Index: 884

Text #2

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.7038. Min Act: -0.0000

Data Index: 309768 (C4 (Web Text))

Max Activating Token Index: 933

Text #3

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6434. Min Act: -0.0000

Data Index: 525728 (C4 (Web Text))

Max Activating Token Index: 932

Text #4

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6536. Min Act: -0.0000

Data Index: 1117068 (C4 (Web Text))

Max Activating Token Index: 384

Text #5

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6384. Min Act: -0.0000

Data Index: 1355463 (C4 (Web Text))

Max Activating Token Index: 408

Text #6

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6349. Min Act: -0.0000

Data Index: 748015 (C4 (Web Text))

Max Activating Token Index: 698

Text #7

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.8102. Min Act: -0.0000

Data Index: 676143 (C4 (Web Text))

Max Activating Token Index: 266

Text #8

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6892. Min Act: -0.0000

Data Index: 96245 (C4 (Web Text))

Max Activating Token Index: 921

Text #9

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6165. Min Act: -0.0000

Data Index: 702105 (C4 (Web Text))

Max Activating Token Index: 936

Text #10

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6162. Min Act: -0.0000

Data Index: 525177 (C4 (Web Text))

Max Activating Token Index: 123

Text #11

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6212. Min Act: -0.0000

Data Index: 208355 (C4 (Web Text))

Max Activating Token Index: 397

Text #12

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.5809. Min Act: -0.0000

Data Index: 670593 (C4 (Web Text))

Max Activating Token Index: 51

Text #13

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6587. Min Act: -0.0000

Data Index: 989345 (C4 (Web Text))

Max Activating Token Index: 632

Text #14

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6025. Min Act: -0.0000

Data Index: 1263895 (C4 (Web Text))

Max Activating Token Index: 795

Text #15

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.5799. Min Act: -0.0000

Data Index: 127987 (C4 (Web Text))

Max Activating Token Index: 995

Text #16

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.5257. Min Act: -0.0000

Data Index: 1181355 (C4 (Web Text))

Max Activating Token Index: 172

Text #17

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.6180. Min Act: -0.0000

Data Index: 255302 (C4 (Web Text))

Max Activating Token Index: 749

Text #18

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.5296. Min Act: -0.0000

Data Index: 535205 (C4 (Web Text))

Max Activating Token Index: 524

Text #19

Max Range: 1.0424. Min Range: -1.0424

Max Act: 0.5122. Min Act: -0.0000

Data Index: 833400 (C4 (Web Text))

Max Activating Token Index: 918