Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 879 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6554. Min Act: -0.0000

Data Index: 686567 (C4 (Web Text))

Max Activating Token Index: 967

Text #1

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6509. Min Act: -0.0000

Data Index: 107951 (C4 (Web Text))

Max Activating Token Index: 514

Text #2

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6501. Min Act: -0.0000

Data Index: 1110465 (C4 (Web Text))

Max Activating Token Index: 349

Text #3

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6691. Min Act: -0.0000

Data Index: 1358272 (C4 (Web Text))

Max Activating Token Index: 815

Text #4

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6577. Min Act: -0.0000

Data Index: 325494 (C4 (Web Text))

Max Activating Token Index: 433

Text #5

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5998. Min Act: -0.0000

Data Index: 261476 (C4 (Web Text))

Max Activating Token Index: 834

Text #6

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6465. Min Act: -0.0000

Data Index: 1321812 (C4 (Web Text))

Max Activating Token Index: 680

Text #7

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6128. Min Act: -0.0000

Data Index: 6820 (C4 (Web Text))

Max Activating Token Index: 531

Text #8

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5880. Min Act: -0.0000

Data Index: 1125429 (C4 (Web Text))

Max Activating Token Index: 222

Text #9

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5989. Min Act: -0.0000

Data Index: 1329071 (C4 (Web Text))

Max Activating Token Index: 524

Text #10

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5885. Min Act: -0.0000

Data Index: 175936 (C4 (Web Text))

Max Activating Token Index: 974

Text #11

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6268. Min Act: -0.0000

Data Index: 65931 (C4 (Web Text))

Max Activating Token Index: 671

Text #12

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5914. Min Act: -0.0000

Data Index: 110956 (C4 (Web Text))

Max Activating Token Index: 967

Text #13

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6141. Min Act: -0.0000

Data Index: 509439 (C4 (Web Text))

Max Activating Token Index: 589

Text #14

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6103. Min Act: -0.0000

Data Index: 1072669 (C4 (Web Text))

Max Activating Token Index: 259

Text #15

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6028. Min Act: -0.0000

Data Index: 459082 (C4 (Web Text))

Max Activating Token Index: 496

Text #16

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.6352. Min Act: -0.0000

Data Index: 1322734 (C4 (Web Text))

Max Activating Token Index: 897

Text #17

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5882. Min Act: -0.0000

Data Index: 784331 (C4 (Web Text))

Max Activating Token Index: 150

Text #18

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5871. Min Act: -0.0000

Data Index: 14922 (C4 (Web Text))

Max Activating Token Index: 767

Text #19

Max Range: 0.6691. Min Range: -0.6691

Max Act: 0.5602. Min Act: -0.0000

Data Index: 1066105 (C4 (Web Text))

Max Activating Token Index: 769