Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2900 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1931. Min Range: -2.1931

Max Act: 2.1931. Min Act: -0.0000

Data Index: 995614 (C4 (Web Text))

Max Activating Token Index: 330

Text #1

Max Range: 2.1931. Min Range: -2.1931

Max Act: 2.0884. Min Act: -0.0000

Data Index: 1322646 (C4 (Web Text))

Max Activating Token Index: 198

Text #2

Max Range: 2.1931. Min Range: -2.1931

Max Act: 2.0163. Min Act: -0.0000

Data Index: 1004053 (C4 (Web Text))

Max Activating Token Index: 809

Text #3

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.9015. Min Act: -0.0000

Data Index: 593936 (C4 (Web Text))

Max Activating Token Index: 588

Text #4

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.8264. Min Act: 0.0000

Data Index: 364849 (C4 (Web Text))

Max Activating Token Index: 609

Text #5

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.7166. Min Act: -0.0000

Data Index: 1247702 (C4 (Web Text))

Max Activating Token Index: 327

Text #6

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.5376. Min Act: -0.0000

Data Index: 1114531 (C4 (Web Text))

Max Activating Token Index: 729

Text #7

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.5398. Min Act: 0.0003

Data Index: 910261 (C4 (Web Text))

Max Activating Token Index: 887

Text #8

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4887. Min Act: -0.0000

Data Index: 1052980 (C4 (Web Text))

Max Activating Token Index: 692

Text #9

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4693. Min Act: -0.0000

Data Index: 1144730 (C4 (Web Text))

Max Activating Token Index: 422

Text #10

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4538. Min Act: -0.0000

Data Index: 1087397 (C4 (Web Text))

Max Activating Token Index: 887

Text #11

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4420. Min Act: -0.0000

Data Index: 1121315 (C4 (Web Text))

Max Activating Token Index: 633

Text #12

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4245. Min Act: -0.0000

Data Index: 211518 (C4 (Web Text))

Max Activating Token Index: 678

Text #13

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4116. Min Act: 0.0000

Data Index: 1293199 (C4 (Web Text))

Max Activating Token Index: 1011

Text #14

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.4617. Min Act: -0.0000

Data Index: 383122 (C4 (Web Text))

Max Activating Token Index: 839

Text #15

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.3958. Min Act: -0.0000

Data Index: 703274 (C4 (Web Text))

Max Activating Token Index: 492

Text #16

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.3369. Min Act: -0.0000

Data Index: 678154 (C4 (Web Text))

Max Activating Token Index: 681

Text #17

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.3255. Min Act: -0.0000

Data Index: 1137924 (C4 (Web Text))

Max Activating Token Index: 533

Text #18

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.3219. Min Act: -0.0000

Data Index: 614273 (C4 (Web Text))

Max Activating Token Index: 397

Text #19

Max Range: 2.1931. Min Range: -2.1931

Max Act: 1.3162. Min Act: 0.0000

Data Index: 395528 (C4 (Web Text))

Max Activating Token Index: 197