Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 492 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.8263. Min Act: -0.0000

Data Index: 76431 (C4 (Web Text))

Max Activating Token Index: 256

Text #1

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.8273. Min Act: -0.0000

Data Index: 750409 (C4 (Web Text))

Max Activating Token Index: 428

Text #2

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.6441. Min Act: -0.0000

Data Index: 371951 (C4 (Web Text))

Max Activating Token Index: 149

Text #3

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.5597. Min Act: -0.0000

Data Index: 1322568 (C4 (Web Text))

Max Activating Token Index: 933

Text #4

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.7156. Min Act: -0.0000

Data Index: 1117790 (C4 (Web Text))

Max Activating Token Index: 148

Text #5

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.6143. Min Act: -0.0000

Data Index: 638622 (C4 (Web Text))

Max Activating Token Index: 633

Text #6

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.4799. Min Act: -0.0000

Data Index: 1157216 (C4 (Web Text))

Max Activating Token Index: 662

Text #7

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.4586. Min Act: -0.0000

Data Index: 524099 (C4 (Web Text))

Max Activating Token Index: 144

Text #8

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.3419. Min Act: -0.0000

Data Index: 80037 (C4 (Web Text))

Max Activating Token Index: 941

Text #9

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.3607. Min Act: -0.0000

Data Index: 1145425 (C4 (Web Text))

Max Activating Token Index: 592

Text #10

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.3381. Min Act: -0.0000

Data Index: 432940 (C4 (Web Text))

Max Activating Token Index: 379

Text #11

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.2397. Min Act: -0.0000

Data Index: 648493 (C4 (Web Text))

Max Activating Token Index: 702

Text #12

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.2877. Min Act: -0.0000

Data Index: 1064957 (C4 (Web Text))

Max Activating Token Index: 567

Text #13

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.3512. Min Act: -0.0000

Data Index: 208122 (C4 (Web Text))

Max Activating Token Index: 731

Text #14

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.2745. Min Act: -0.0000

Data Index: 810599 (C4 (Web Text))

Max Activating Token Index: 896

Text #15

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.1952. Min Act: -0.0000

Data Index: 66713 (C4 (Web Text))

Max Activating Token Index: 177

Text #16

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.3304. Min Act: -0.0000

Data Index: 300072 (C4 (Web Text))

Max Activating Token Index: 289

Text #17

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.1718. Min Act: -0.0000

Data Index: 696397 (C4 (Web Text))

Max Activating Token Index: 135

Text #18

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.1712. Min Act: -0.0000

Data Index: 512880 (C4 (Web Text))

Max Activating Token Index: 244

Text #19

Max Range: 1.8273. Min Range: -1.8273

Max Act: 1.1591. Min Act: -0.0000

Data Index: 125825 (C4 (Web Text))

Max Activating Token Index: 434