Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 44 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.4500. Min Act: -0.0000

Data Index: 231163 (C4 (Web Text))

Max Activating Token Index: 897

Text #1

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.2601. Min Act: -0.0000

Data Index: 952805 (C4 (Web Text))

Max Activating Token Index: 868

Text #2

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.3258. Min Act: -0.0000

Data Index: 138546 (C4 (Web Text))

Max Activating Token Index: 245

Text #3

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.3427. Min Act: -0.0000

Data Index: 1144585 (C4 (Web Text))

Max Activating Token Index: 604

Text #4

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.3935. Min Act: -0.0000

Data Index: 1133163 (C4 (Web Text))

Max Activating Token Index: 551

Text #5

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.2186. Min Act: -0.0000

Data Index: 78050 (C4 (Web Text))

Max Activating Token Index: 267

Text #6

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.2544. Min Act: -0.0000

Data Index: 254690 (C4 (Web Text))

Max Activating Token Index: 525

Text #7

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.2009. Min Act: -0.0000

Data Index: 1342959 (C4 (Web Text))

Max Activating Token Index: 197

Text #8

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.1824. Min Act: -0.0000

Data Index: 830297 (C4 (Web Text))

Max Activating Token Index: 703

Text #9

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.1824. Min Act: -0.0000

Data Index: 535923 (C4 (Web Text))

Max Activating Token Index: 734

Text #10

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0432. Min Act: -0.0000

Data Index: 601691 (C4 (Web Text))

Max Activating Token Index: 844

Text #11

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0961. Min Act: -0.0000

Data Index: 1023257 (C4 (Web Text))

Max Activating Token Index: 601

Text #12

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.1397. Min Act: -0.0000

Data Index: 512255 (C4 (Web Text))

Max Activating Token Index: 604

Text #13

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0715. Min Act: -0.0000

Data Index: 1199291 (C4 (Web Text))

Max Activating Token Index: 619

Text #14

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0718. Min Act: -0.0000

Data Index: 341350 (C4 (Web Text))

Max Activating Token Index: 740

Text #15

Max Range: 1.4500. Min Range: -1.4500

Max Act: 0.9813. Min Act: -0.0000

Data Index: 1040677 (C4 (Web Text))

Max Activating Token Index: 826

Text #16

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0200. Min Act: -0.0000

Data Index: 6253 (C4 (Web Text))

Max Activating Token Index: 640

Text #17

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0338. Min Act: -0.0000

Data Index: 844675 (C4 (Web Text))

Max Activating Token Index: 724

Text #18

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0459. Min Act: -0.0000

Data Index: 594040 (C4 (Web Text))

Max Activating Token Index: 233

Text #19

Max Range: 1.4500. Min Range: -1.4500

Max Act: 1.0829. Min Act: -0.0000

Data Index: 1243540 (C4 (Web Text))

Max Activating Token Index: 312