Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3762 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.1063. Min Act: -0.0000

Data Index: 321365 (C4 (Web Text))

Max Activating Token Index: 968

Text #1

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.1508. Min Act: -0.0000

Data Index: 472864 (C4 (Web Text))

Max Activating Token Index: 440

Text #2

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.0556. Min Act: -0.0000

Data Index: 735411 (C4 (Web Text))

Max Activating Token Index: 833

Text #3

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.0270. Min Act: -0.0000

Data Index: 920213 (C4 (Web Text))

Max Activating Token Index: 89

Text #4

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.0303. Min Act: -0.0000

Data Index: 236912 (C4 (Web Text))

Max Activating Token Index: 781

Text #5

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.9676. Min Act: -0.0000

Data Index: 1019890 (C4 (Web Text))

Max Activating Token Index: 472

Text #6

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8896. Min Act: -0.0000

Data Index: 840143 (C4 (Web Text))

Max Activating Token Index: 684

Text #7

Max Range: 1.1508. Min Range: -1.1508

Max Act: 1.0011. Min Act: -0.0000

Data Index: 562825 (C4 (Web Text))

Max Activating Token Index: 795

Text #8

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.9317. Min Act: -0.0000

Data Index: 647321 (C4 (Web Text))

Max Activating Token Index: 540

Text #9

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8783. Min Act: -0.0000

Data Index: 1181930 (C4 (Web Text))

Max Activating Token Index: 441

Text #10

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8416. Min Act: -0.0000

Data Index: 759354 (C4 (Web Text))

Max Activating Token Index: 121

Text #11

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8688. Min Act: -0.0000

Data Index: 958813 (C4 (Web Text))

Max Activating Token Index: 742

Text #12

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8935. Min Act: -0.0000

Data Index: 1139992 (C4 (Web Text))

Max Activating Token Index: 62

Text #13

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8586. Min Act: -0.0000

Data Index: 791761 (C4 (Web Text))

Max Activating Token Index: 455

Text #14

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8551. Min Act: -0.0000

Data Index: 898710 (C4 (Web Text))

Max Activating Token Index: 71

Text #15

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8465. Min Act: -0.0000

Data Index: 1315634 (C4 (Web Text))

Max Activating Token Index: 807

Text #16

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8928. Min Act: -0.0000

Data Index: 731822 (C4 (Web Text))

Max Activating Token Index: 707

Text #17

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8871. Min Act: -0.0000

Data Index: 397423 (C4 (Web Text))

Max Activating Token Index: 1013

Text #18

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.7934. Min Act: -0.0000

Data Index: 459132 (C4 (Web Text))

Max Activating Token Index: 795

Text #19

Max Range: 1.1508. Min Range: -1.1508

Max Act: 0.8374. Min Act: -0.0000

Data Index: 1080075 (C4 (Web Text))

Max Activating Token Index: 726