Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 213 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0607. Min Act: -0.0002

Data Index: 386671 (C4 (Web Text))

Max Activating Token Index: 64

Text #1

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0597. Min Act: -0.0001

Data Index: 1090283 (C4 (Web Text))

Max Activating Token Index: 108

Text #2

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0582. Min Act: -0.0001

Data Index: 1319202 (C4 (Web Text))

Max Activating Token Index: 6

Text #3

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0569. Min Act: -0.0001

Data Index: 682063 (C4 (Web Text))

Max Activating Token Index: 996

Text #4

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0581. Min Act: -0.0001

Data Index: 703803 (C4 (Web Text))

Max Activating Token Index: 169

Text #5

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0572. Min Act: -0.0001

Data Index: 498540 (C4 (Web Text))

Max Activating Token Index: 827

Text #6

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0554. Min Act: -0.0001

Data Index: 957048 (C4 (Web Text))

Max Activating Token Index: 80

Text #7

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0554. Min Act: -0.0001

Data Index: 857552 (C4 (Web Text))

Max Activating Token Index: 65

Text #8

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0538. Min Act: -0.0001

Data Index: 448404 (C4 (Web Text))

Max Activating Token Index: 1001

Text #9

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0531. Min Act: -0.0001

Data Index: 477779 (C4 (Web Text))

Max Activating Token Index: 6

Text #10

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0539. Min Act: -0.0001

Data Index: 121434 (C4 (Web Text))

Max Activating Token Index: 200

Text #11

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0522. Min Act: -0.0001

Data Index: 354765 (C4 (Web Text))

Max Activating Token Index: 758

Text #12

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0537. Min Act: -0.0001

Data Index: 777069 (C4 (Web Text))

Max Activating Token Index: 612

Text #13

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0528. Min Act: -0.0001

Data Index: 901247 (C4 (Web Text))

Max Activating Token Index: 57

Text #14

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0519. Min Act: -0.0001

Data Index: 869637 (C4 (Web Text))

Max Activating Token Index: 843

Text #15

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0523. Min Act: -0.0001

Data Index: 1268348 (C4 (Web Text))

Max Activating Token Index: 801

Text #16

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0510. Min Act: -0.0001

Data Index: 245451 (C4 (Web Text))

Max Activating Token Index: 329

Text #17

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0507. Min Act: -0.0001

Data Index: 617893 (C4 (Web Text))

Max Activating Token Index: 94

Text #18

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0522. Min Act: -0.0001

Data Index: 801088 (C4 (Web Text))

Max Activating Token Index: 337

Text #19

Max Range: 0.0607. Min Range: -0.0607

Max Act: 0.0503. Min Act: -0.0001

Data Index: 467253 (C4 (Web Text))

Max Activating Token Index: 728